Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dns processor - Add A, AAAA, and TXT query support #36394

Merged

Conversation

andrewkroh
Copy link
Member

@andrewkroh andrewkroh commented Aug 22, 2023

Proposed commit message

The dns processor previously supported only reverse DNS lookups.
This adds support for performing A, AAAA, and TXT record queries.

The response.ptr.histogram metric was renamed to request_duration.histogram.
This naming allows the metric to represent the duration of the DNS request
for all query types.

Closes #11416

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Examples

filebeat.yml:

---

filebeat.inputs:
  - type: cel
    publisher_pipeline:
      disable_host: true
    interval: 1m
    resource:
      url: not_used_1
    redact:
      fields: ~
    program: |
      {
          'want_more': false,
          'events': [
              {
                'message': 'newyork.weather',
              }
          ],
      }
    processors:
      - dns:
          type: TXT
          nameservers: ['dns.toys']
          fields:
            message: weather

  - type: cel
    publisher_pipeline:
      disable_host: true
    interval: 1m
    resource:
      url: not_used_2
    redact:
      fields: ~
    program: |
      {
          'want_more': false,
          'events': [
              {
                  'host': {
                      'hostname': 'www.elastic.co'
                  }
              }
          ],
      }
    processors:
      - dns:
          type: A
          nameservers: ['8.8.8.8']
          fields:
            host.hostname: host.ip

  - type: cel
    publisher_pipeline:
      disable_host: true
    interval: 1m
    resource:
      url: not_used_3
    redact:
      fields: ~
    program: |
      {
          'want_more': false,
          'events': [
              {
                  'host': {
                      'hostname': 'ipv6.google.com'
                  }
              }
          ],
      }
    processors:
      - dns:
          type: AAAA
          nameservers: ['8.8.8.8']
          fields:
            host.hostname: host.ip

processors:
  - drop_fields:
      ignore_missing: true
      fields:
        - agent
        - ecs
        - input

output.console.enabled: true

http:
  host: localhost
  port: 5066

output events:

{
  "@timestamp": "2023-08-22T13:57:30.641Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.11.0"
  },
  "host": {
    "hostname": "www.elastic.co",
    "ip": [
      "151.101.2.217",
      "151.101.66.217",
      "151.101.130.217",
      "151.101.194.217"
    ]
  }
}
{
  "@timestamp": "2023-08-22T13:57:30.639Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.11.0"
  },
  "weather": [
    "New York (US)",
    "21.00C (69.80F)",
    "60.60% hu.",
    "partlycloudy_day",
    "10:00, Tue",
    "New York (US)",
    "23.70C (74.66F)",
    "51.60% hu.",
    "cloudy",
    "12:00, Tue",
    "New York (US)",
    "25.40C (77.72F)",
    "46.30% hu.",
    "partlycloudy_day",
    "14:00, Tue",
    "New York (US)",
    "26.20C (79.16F)",
    "42.90% hu.",
    "fair_day",
    "16:00, Tue",
    "New York (US)",
    "25.50C (77.90F)",
    "42.70% hu.",
    "cloudy",
    "18:00, Tue"
  ],
  "message": "newyork.weather"
}
{
  "@timestamp": "2023-08-22T13:57:30.641Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "8.11.0"
  },
  "host": {
    "hostname": "ipv6.google.com",
    "ip": [
      "2607:f8b0:4004:c07::64",
      "2607:f8b0:4004:c07::66",
      "2607:f8b0:4004:c07::8a",
      "2607:f8b0:4004:c07::71"
    ]
  }
}

% jq .processor dns-stats.json

{
  "dns": {
    "1": {
      "cache": {
        "hits": 0,
        "misses": 1
      },
      "dns_toys": {
        "failure": 0,
        "request_duration": {
          "histogram": {
            "count": 1,
            "max": 19227625,
            "mean": 19227625,
            "median": 19227625,
            "min": 19227625,
            "p75": 19227625,
            "p95": 19227625,
            "p99": 19227625,
            "p999": 19227625,
            "stddev": 0
          }
        },
        "success": 1
      }
    },
    "2": {
      "8_8_8_8": {
        "failure": 0,
        "request_duration": {
          "histogram": {
            "count": 1,
            "max": 8290708,
            "mean": 8290708,
            "median": 8290708,
            "min": 8290708,
            "p75": 8290708,
            "p95": 8290708,
            "p99": 8290708,
            "p999": 8290708,
            "stddev": 0
          }
        },
        "success": 1
      },
      "cache": {
        "hits": 0,
        "misses": 1
      }
    },
    "3": {
      "8_8_8_8": {
        "failure": 0,
        "request_duration": {
          "histogram": {
            "count": 1,
            "max": 19198583,
            "mean": 19198583,
            "median": 19198583,
            "min": 19198583,
            "p75": 19198583,
            "p95": 19198583,
            "p99": 19198583,
            "p999": 19198583,
            "stddev": 0
          }
        },
        "success": 1
      },
      "cache": {
        "hits": 0,
        "misses": 1
      }
    }
  }
}

Sorry, something went wrong.

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 22, 2023
@mergify
Copy link
Contributor

mergify bot commented Aug 22, 2023

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @andrewkroh? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

@elasticmachine
Copy link
Collaborator

elasticmachine commented Aug 22, 2023

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-08-23T13:55:56.903+0000

  • Duration: 108 min 3 sec

Test stats 🧪

Test Results
Failed 0
Passed 28073
Skipped 2015
Total 30088

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

The dns processor previously supported only reverse DNS lookups.
This adds support for performing A, AAAA, and TXT record queries.

The response.ptr.histogram metric was renamed to request_duration.histogram.
This naming allows the metric to represent the duration of the DNS request
for all query types.

Closes elastic#11416
@andrewkroh andrewkroh force-pushed the feature/libbeat/dns-forward-lookup branch from 4598a67 to 6ccdfdf Compare August 22, 2023 21:42
@andrewkroh andrewkroh force-pushed the feature/libbeat/dns-forward-lookup branch from 6ccdfdf to 64b7032 Compare August 22, 2023 22:05
@andrewkroh andrewkroh marked this pull request as ready for review August 22, 2023 22:28
@andrewkroh andrewkroh requested a review from a team as a code owner August 22, 2023 22:28
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

libbeat/processors/dns/config.go Outdated Show resolved Hide resolved

// Unpack unpacks a string to a queryType.
func (qt *queryType) Unpack(v string) error {
switch strings.ToLower(v) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe use strings.EqualFold? It's more verbose, so up to you.

libbeat/processors/dns/config.go Outdated Show resolved Hide resolved
libbeat/processors/dns/dns.go Outdated Show resolved Hide resolved
libbeat/processors/dns/dns.go Outdated Show resolved Hide resolved
libbeat/processors/dns/dns.go Outdated Show resolved Hide resolved
libbeat/processors/dns/resolver.go Outdated Show resolved Hide resolved
libbeat/processors/dns/resolver.go Outdated Show resolved Hide resolved
@andrewkroh andrewkroh added backport-skip Skip notification from the automated backport with mergify 8.11-candidate labels Aug 23, 2023
@andrewkroh andrewkroh merged commit 3161fc0 into elastic:main Aug 25, 2023
Scholar-Li pushed a commit to Scholar-Li/beats that referenced this pull request Feb 5, 2024
The dns processor previously supported only reverse DNS lookups.
This adds support for performing A, AAAA, and TXT record queries.

The response.ptr.histogram metric was renamed to request_duration.histogram.
This naming allows the metric to represent the duration of the DNS request
for all query types.

Some refactoring was done to unexport types/functions that should have been
internal only.

Closes elastic#11416

Co-authored-by: Dan Kortschak <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
8.11-candidate backport-skip Skip notification from the automated backport with mergify enhancement libbeat :Processors
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature - Resolve DNS Enrichment (Forward Lookup)
3 participants